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1  Overview 


This  project  focused  on  developing  advanced  security  technology  in  the  areas  of  operating  systems,  software 
components,  distributed  shared  memory,  and  language  types.  The  security  emphasis  of  much  of  the  work  was 
on  resource  management.  As  originally  conceived  and  proposed  in  April  1995,  the  software  base  for  the  OS 
work  was  to  be  Utah’s  version  of  the  Mach  kernel  and  operating  system,  the  component  work  was  to  build 
on  the  Flick  IDL  compiler  and  Utah’s  OMOS  object  server,  and  the  distributed  shared  memory  work  was  to 
be  an  extension  of  Utah’s  nascent  Quarks  system,  oriented  to  closely  coupled  cluster  environments. 

However,  the  grant  did  not  actually  begin  until  over  a  year  later,  in  August  1996.  By  then  we  had  maHp 
progress  that  caused  some  of  our  research  infrastructure  and  our  research  emphases  to  change. 

With  DARPA’s  approval,  in  late  1995  we  moved  away  from  the  Mach  kernel  as  an  OS  development  base 
to  Fluke,  a  new  operating  system  architecture  and  microkernel  that  we  developed.  For  component  work  we 
moved  from  OMOS  to  the  OSKiv,  the  Hick  IDL  compiler  remained  an  important  research  base,  as  originally 
proposed.  We  also  moved  our  distributed  shared  memory  focus  from  the  local  area  to  the  more  general  and, 
we  believe,  more  important  problem  of  providing  infrastructure  to  a  wide  variety  of  distributed  applications 
and  over  a  wider  geographical  area  than  a  cluster.  Out  of  this  grew  the  Khazana/KOLA  system.  Finally,  we 
applied  language  type-based  protection  to  developing  Java  operating  systems,  replying  our  OS  structuring 
techniques  to  Java  virtual  machines. 

These  changes  set  the  context  for  all  of  the  work  performed  under  this  grant.  We  now  summarize  original 
and  new  approaches  in  each  of  the  areas  of  work  in  the  original  proposal,  plus  three  entirely  new  areas. 

•  Mach  kernel  and  OS:  resource  management  and  information  control 

New  Fluke  kernel  and  OS:  new  infrastructure,  same  topic 

•  Adaptation 

Replaced  by  adaptive  handling  of  network  bandwidth:  OSKit  hpfq  component 

•  Flick  IDL  Compiler 

Same 

•  Secure  local  DSM 

Khazana/KOLA:  Wide-area  caching  infrastructure 

•  Language  type-based  security 

Language  type-based  (Java)  OS  implementations:  Alta.  GVM 

•  Application  to  test  cases 

Secure  environments  ( wrappers),  plus  design  of  sophisticated  applications  (NRL  pump ) 

•  Official  collaboration 

NS  A:  very  close;  WUSTL,  BBN,  CMU:  moderate 

•  Entirely  new  areas,  not  originally  proposed: 

—  Flask  high-security  kernel  and  OS 
—  User-level  device  drivers 
-  Formal  methods  applied  to  Fluke  IPC 
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Scope:  Approximately  2.5  years  of  this  3  year  project  overlapped  with  another  DARPA-funded  project 
of  ours,  “Fast  and  Flexible  Mach-Based  Systems,”  AO  #B799,  monitored  by  the  Department  of  the  Army. 
As  discussed  with  our  Rome  Air  Force  Research  Lab  grant  monitors  and  DARPA  Program  Manager  during 
their  visit  to  Utah  on  18  March  1998,  the  extremely  intertwined  nature  of  these  projects  made  it  impossible  to 
separate  and  allocate  tasks  entirely  to  one  contract  or  the  other.  Therefore  they  approved  having  our  quarterly 
reports  partially  overlap  in  content.  We  follow  the  same  procedure  in  this  report:  results  that  were  achieved 
before  or  after  the  overlapping  2.5  years  are  reported  only  in  the  appropriate  report;  during  the  overlap  results 
are  attributed  to  both  grants.  A  few  of  the  papers  show  a  publication  date  beyond  the  end  of  the  respective 
grant  period;  in  these  cases  the  reported  work  and  all  or  nearly  all  of  the  writing  was  actually  accomplished 
during  the  grant  period.  Normal  latency  in  the  publishing  process  accounts  for  the  apparent  discrepancy. 

Administrative  notes:  We  had  an  unusually  low  rate  of  expenditures  during  the  early  years  of  this  project. 
The  most  important  reason  for  this  was  the  project’s  technical  overlap  with  the  D ARPA-funded  project  men¬ 
tioned  above,  combined  with  the  scarcity  of  “systems”  students  and  qualified  staff  due  to  high  demand  by 
industry,  compounded,  for  staff,  by  University-mandated  pay  scales.  (With  the  current  Silicon  Valley  indus¬ 
trial  boom,  this  problem  will  just  get  more  severe.)  However,  much  of  the  work  was  still  carried  out  under  the 
other  DARPA-funded  project.  That  project  could  not  find  enough  staff  either,  so  the  pace  of  progress  in  both 
projects  was  slower  than  planned.  Since  no  extension  of  this  grant’s  duration  was  possible,  the  low  expendi¬ 
ture  rate  in  the  grant’s  early  years  caused  the  total  funding  received  under  this  grant  to  be  only  $1,742,987, 
not  the  originally  planned  amount  of  $2,542,987.  This  31%  reduction  in  funding  did  cause  some  reduction 
in  our  planned  accomplishments,  but  nevertheless,  we  did  deliver  a  great  number  of  results,  both  in  software 
and  publications. 

Outline:  The  bulk  of  the  rest  of  this  report  is  structured  around  the  many  software  systems  developed  under 
this  project.  They  fall  into  six  categories: 

1.  The  Fluke/Flask  Kernel  and  Operating  System 

2.  The  OSKit:  Reusable  Components  for  OS  Implementation 

3.  The  Flick  Interface  Definition  Language  Compiler 

4.  MOM:  The  Mini  Object  Model 

5.  Java  Operating  Systems 

6.  The  Khazana/KOLA  Infrastructure  for  Building  Distributed  Services 

This  is  followed  by  some  comments  on  technology  transfer  in  Section  3.  The  report  concludes  with  a 
categorized  list  of  publications. 
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2  Results 

2.1  The  Fluke/Flask  Kernel  and  Operating  System 

Background:  In  mid-December  1995  we  met  with  Gary  Koob,  our  DARPA  program  manager  for  the  re¬ 
lated  contract  (AO  #B799),  and  discussed  a  change  in  strategy  to  achieve  that  project’s  original  goals,  as  well 
as  to  attempt  more  ambitious  ones.  In  the  next  quarter  that  proposed  direction  crystallized  and  work  accel¬ 
erated  in  that  direction.  Teresa  Lunt  (at  the  time  our  DARPA  program  manager  for  this  grant),  Gary  Koob, 
Bob  Parker,  and  Jeff  Turner  (NCSC/NSA),  all  seemed  pleased  with  our  new  strategy,  and  we  were  gratified 
both  by  the  response  and  our  own  progress. 

In  summary,  before  this  grant  began,  we  moved  away  from  the  Mach  kernel  to  Fluke ,  a  novel  operat¬ 
ing  system  structure  designed  for  high  assurance  and  efficient  support  of  recursive  virtual  machines.1  We 
conceived  the  new  structure,  developed  the  requirements  for  the  new  kernel,  and  designed  the  kernel  archi¬ 
tecture. 

During  the  term  of  this  grant,  we  worked  closely  with  the  National  Security  Agency  and  Secure  Com¬ 
puting  Corporation  to  design  and  implement  additions  to  Fluke  to  form  a  high-security  system  railed  Flask. 
We  implemented  a  prototype  Fluke/Flask  kernel  on  the  Intel  x86,  implemented  ten  virtual  machine  monitors 
and  servers,  including  a  virtual  memory  manager,  a  checkpointer,  a  process  manager,  a  file  server,  a  network 
server,  a  secure  network  server,  and  a  server  providing  transparent  distributed  IPC.  Several  libraries  to  sup¬ 
port  these  services  were  also  designed  and  implemented.  The  overall  Fluke  operating  system  includes  major 
additional  components  which  we  developed  and  separately  packaged  and  distributed:  the  OSKit,  the  Flick 
IDL  compiler,  and  the  Mini  Object  Model  and  runtime.  These  are  described  in  later  sections. 

Our  Fluke/Flask  kernel  implementation  was  designed  to  be  relatively  simple  in  order  to  facilitate  exper¬ 
imentation,  to  be  as  portable  as  possible,  to  provide  maximum  error  checking  and  reporting,  to  conform  to 
strict  code  modularity  and  layering  disciplines  in  anticipation  of  high-security  implementations,  and  to  reuse 
as  much  existing  code  as  possible.  For  all  of  these  reasons  this  implementation  of  the  Fluke  kernel  architec¬ 
ture  is  not  optimized,  and  conclusions  regarding  the  performance  limits  of  the  Fluke  architecture  should  not 
be  drawn  from  it.  Application  of  the  well-documented  design  and  implementation  techniques  demonstrated 
by  kernels  such  as  L4  would  markedly  improve  Fluke’s  performance. 

Fluke  Motivation:  the  Persistent  Relevance  of  the  Local  Operating  System:  The  growth  and  popular¬ 
ity  of  loosely-coupled  distributed  systems  such  as  the  World  Wide  Web  and  the  touting  of  Java-based  systems 
as  the  solution  to  the  issues  of  software  maintenance,  flexibility,  and  security  are  changing  the  research  em¬ 
phasis  away  from  traditional  single  node  operating  system  issues.  Apparently,  the  view  is  that  traditional  OS 
issues  are  either  solved  problems  or  minor  problems.  By  contrast,  we  believed  that  building  such  vast  dis¬ 
tributed  systems  upon  the  fragile  infrastructure  provided  by  today’s  operating  systems  is  analogous  to  build¬ 
ing  castles  on  sand.  In  a  paper  [1]  at  a  high-visibility  international  workshop  we  outlined  the  supporting  ar¬ 
guments  for  these  views  and  described  the  Fluke  OS  design  that  supports  secure  encapsulation  of  the  foreign 
processes  that  will  be  increasingly  prevalent  in  tomorrow’s  distributed  systems. 

'It  is  important  to  note  that  Fluke's  “virtual  machines"  have  nothing  inherently  to  do  with  Java.  They  are  not  byte  code  inter¬ 
preters.  but  are  more  akin  to  the  “virtual  machine  monitors"  of  classical  virtual  machine  architectures  of  the  I970*s. 
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2.1.1  Fluke/Flask  OS  Components 


The  Fluke/Flask  operating  system  is  highly  modular  and  highly  structured,  as  is  appropriate  for  a  high-security 
or  safety-critical  system.  Besides  the  kernel,  the  Fluke  OS  includes  a  large  number  of  servers,  libraries,  and 
IDL-specified  interfaces  between  processes.  Most  of  the  major  servers  providing  POSIX-like  function  (e.g., 
network,  filesystems)  are  themselves  highly  modular,  containing  unchanged  OSKit  components. 

•  “Common  protocols”:  IPC-based  interfaces  between  the  virtual  machine  layers  (between  the  “nesters”), 
specified  in  CORB  A IDL.  For  example,  the  common  protocols  include  interfaces  for  memory  alloca¬ 
tion,  networking,  file  access,  and  device  access. 

•  Fluke  libraries 

-  libsac:  Provides  a  POSIX-like  interface  to  client  applications,  including  optimized  access  to 
Fluke  IPC-based  services. 

-  libnest:  Common  infrastructure  for  Fluke  servers  (nesters),  such  as  functions  to  create  and  ini¬ 
tialize  new  virtual  environments  (new  levels  of  nesting). 

-  libmm :  Memory  management.  Used  by  both  the  kernel  and  the  virtual  memory  manager. 

-  libwrap:  Used  to  interpose  on  the  common  protocols,  to  provide  sandboxing  or  logging. 

-  libmemdebug:  Memory  allocation  debugging  aid. 

-  libyfs:  Simple  filesystem,  useful  for  development. 

•  Fluke  servers 

-  kernel  server:  The  “root”  nester;  this  is  a  normal  Fluke  nester,  except  that  it  has  the  same  memory 
mappings  as  the  kernel.  The  source  of  resources  (e.g.  memory,  device  access). 

-  virtual  memory  manager:  Obtains  physical  memory  from  an  ancestor(normally  the  kernel  server) 
and  provides  virtual  memory  to  all  descendent  processes.  Normally  the  first  nester  started  outside 
the  kernel,  except  in  Flask,  when  the  security  policy  server  is  first. 

-  checkpointer:  Provides  checkpointing  of  descendent  processes. 

-  process  manager:  Provides  process  management  for  its  immediate  child  processes;  e.g.,  fork/kill/signal. 

-  branch  nester:  Forks  a  branch  of  the  virtual  machine  hierarchy  into  two  branches,  without  chang¬ 
ing  anything  else  (unlike  the  process  manager,  above). 

-  logging  nester:  When  interposed  into  a  branch  of  the  virtual  machine  hierarchy,  logs  all  com¬ 
munication  between  levels. 

-  network  server:  Provides  TCP/IP  and  sockets. 

-  network  (distributed)  IPC  server:  Extends  Fluke  IPC  across  a  network. 

-  real  file  server:  Provides  full  BSD  filesystem  functionality. 

-  memory  file  server:  RAM  disk  filesystem. 

-  trivial  file  server:  A  simple  file  server. 

•  Flask-specific  libraries  and  servers 

-  security  policy  server:  Based  on  a  policy  database,  provides  decisions  on  access,  object  labeling, 
and  polyinstantiation.  Normally  the  first  Flask  server  to  be  started. 

-  libavc:  Access  Vector  Cache:  minimizes  and  optimizes  communication  to  the  security  policy 
server,  while  providing  coherence. 
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2.1.2  Recursive  Virtual  Machine  Architecture 

Fluke  brings  together  two  lines  of  research  and  two  heretofore  disparate  architectures:  microkernels  and  re¬ 
cursive  virtual  machines  [2]. 

Fluke  represents  a  novel  approach  to  providing  modular  and  extensible  operating  system  functionality 
and  encapsulated  environments  based  on  a  synthesis. of  microkernel  and  virtual  machine  concepts.  We  de¬ 
veloped  a  software-based  virtualizable  architecture  called  Fluke  that  allows  recursive  virtual  machines  (vir¬ 
tual  machines  running  on  other  virtual  machines)  to  be  implemented  efficiently  by  a  microkernel  running  on 
generic  hardware.  A  complete  virtual  machine  interface  is  provided  at  each  level;  efficiency  derives  from 
needing  to  implement  only  new  functionality  at  each  level.  This  infrastructure  allows  common  OS  function¬ 
ality,  such  as  process  management,  demand  paging,  fault  tolerance,  and  debugging  support,  to  be  provided 
by  cleanly  modularized,  independent,  stackable  virtual  machine  monitors,  implemented  as  user  processes. 
It  can  also  provide  uncommon  or  unique  OS  features,  including  the  above  features  specialized  for  particular 
applications’  needs,  virtual  machines  transparently  distributed  cross-node,  or  security  monitors  that  allow 
arbi  trary  untrusted  binaries  to  be  executed  safely.  Our  prototype  implementation  of  this  model  indicates  that 
it  is  practical  to  modularize  operating  systems  this  way.  Some  types  of  virtual  machine  layers  impose  al¬ 
most  no  overhead  at  all,  while  other  impose  some  overhead  (typically  0-35%),  but  only  on  certain  classes  of 
applications. 


2.13  Flexible  Checkpointing 


One  of  the  novel  features  provided  by  this  fully  “nestable”  OS  architecture  is  the  ability  for  an  otherwise 
ordinary  user  process  to  checkpoint  (or  migrate)  the  state  of  unchanged,  arbitrary  child  processes.  Indeed, 
any  set  of  programs  may  be  checkpointed.  Transparent  checkpointing  of  unchanged  programs  is  itself  highly 
unusual.  As  far  as  we  know  only  three  systems  have  ever  provided  that  feature,  and  they  are  either  old  or 
experimental.  L3  ,  KeyKOS  ,  and  KeyKOS’s  contemporary  re-implementation,  EROS  .  However,  in  those 
OS’s  the  entire  computer  system  is  persistent.  Fluke  is  novel  in  the  flexible  scope  one  can  choose  for  its 
persistence  feature:  anywhere  on  the  spectrum  from  one  (or  zero)  processes,  to  the  whole  machine. 

Such  checkpointing  requires  complete  access  to  the  entire  state  of  a  process,  and  Fluke  is  novel  in  that 
it  can  provide  such  access,  in  a  secure  manner,  to  any  ordinary  user  process  [3].  Checkpointing,  process 
migration,  and  similar  services  need  to  have  access  not  only  to  the  memory  of  the  constituent  processes,  but 
also  to  the  complete  state  of  all  kernel  provided  objects  (e.g.,  threads  and  ports)  involved.  Traditionally,  a 
major  stumbling  block  in  these  operations  is  acquiring  and  re-creating  the  state  in  the  operating  system. 


We  implemented  a  transparent  user-mode  checkpointer  as  an  application  on  the  Fluke  microkernel.  The 
microkernel  consistently  and  cleanly  supports  the  importing  and  exporting  of  fundamental  kernel  state  safely 
to  and  from  user  applications.  Implementing  a  transparent  checkpointing  facility  with  this  sort  of  kernel  sup¬ 
port  simplifies  the  implementation,  and  expands  its  flexibility  and  power. 


2.1.4  Atomicity  of  Kernel  API  and  Kernel  Operations 

In  order  to  provide  completely  exportable  kernel  state  a  novel  kernel  property  is  required.  This  property  re¬ 
lates  to  the  “ atomic ”  nature  of  the  kernel's  application  programming  interface— Ms  API  [4].  In  the  Fluke 
kernel  we  defined  and  implemented  a  kernel  API  that  makes  every  exported  operation  fully  interruptible 
and  restartable,  thereby  appearing  atomic  to  the  user.  To  achieve  interruptibility,  all  possible  kernel  states 
in  which  a  thread  may  become  blocked  for  a  “long”  time  are  represented  as  kernel  system  calls,  without  re¬ 
quiring  the  kernel  to  retain  any  unexposable  internal  state. 
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Since  all  kernel  operations  appear  atomic,  services  such  as  transparent  checkpointing  and  process  mi¬ 
gration  that  need  access  to  the  complete  and  consistent  state  of  a  process  can  be  implemented  by  ordinary 
user-mode  processes.  Atomic  operations  also  enable  applications  to  provide  reliability  in  a  more  straightfor¬ 
ward  manner. 

This  API  also  allowed  us  to  explore  novel  kernel  implementation  techniques  and  to  evaluate  existing  tech- 
niques.  The  Fluke  kernel’s  single  source  implements  either  the  “process”  or  the  “interrupt”  execution  model 
on  both  uniprocessors  and  multiprocessors,  depending  on  a  configuration  option  affecting  a  small  amount  of 
code. 

We  reported  measurements  comparing  fully,  partially  and  non-preemptible  configurations  of  both  process 
and  interrupt  model  implementations.  We  found  that  the  interrupt  model  has  a  modest  performance  advan¬ 
tage  in  some  benchmarks,  maximum  preemption  latency  varies  nearly  three  orders  of  magnitude,  average 
preemption  latency  varies  by  a  factor  of  six,  and  memory  use  favors  the  interrupt  model  as  expected,  but  not 
by  a  large  amount.  We  found  that  the  overhead  for  restarting  the  most  costly  kernel  operation  ranges  from 
2-8%  of  the  cost  of  the  operation. 

This  atomicity  of  the  kernel  interface  provides  an  important  building  block  for  die  high-security  variant 
of  Fluke,  called  “Flask.”  Revocation  of  permissions  is  a  key  challenge  for  secure  systems,  and  as  we’ll  see 
later.  Fluke’s  “atomic”  API  makes  revocation  significantly  more  tractable. 

lli  CPU  Inheritance  Scheduling 

Fluke's  recursive  architecture  is  a  good  match  for  hierarchical  resource  management  schemes,  which  them¬ 
selves  are  a  good  match  to  the  natural  administrative  hierarchy  found  in  the  real  world.  In  designing  re¬ 
source  management  frameworks  for  Fluke,  we  developed  a  novel  hierarchical  processor  scheduling  frame¬ 
work  called  CPU  inheritance  scheduling  [5].  This  is  a  framework  for  scheduling  policies,  not  a  policy  itself, 
and  is  applicable  far  beyond  the  Fluke  kernel. 

Traditional  processor  scheduling  mechanisms  in  operating  systems  are  fairly  rigid,  often  supporting  only 
one  fixed  scheduling  policy,  or,  at  most,  a  few  “scheduling  classes”  whose  implementations  are  closely  tied 
together  in  the  OS  kernel.  We  invented  CPU  inheritance  scheduling,  a  novel  processor  scheduling  frame¬ 
work  in  which  arbitrary  threads  can  act  as  schedulers  for  other  threads.  Widely  different  scheduling  poli¬ 
cies  can  be  implemented  under  the  framework,  and  many  different  policies  can  coexist  in  a  single  system, 
providing  much  greater  scheduling  flexibility.  Modular,  hierarchical  control  can  be  provided  over  the  pro¬ 
cessor  utilization  of  arbitrary  administrative  domains,  such  as  processes,  jobs,  users,  and  groups,  and  the 
CPU  resources  consumed  can  be  accounted  for  and  attributed  accurately.  Applications,  as  well  as  the  OS, 
can  implement  customized  local  scheduling  policies;  the  framework  ensures  that  all  the  different  policies 
work  together  logically  and  predictably.  As  a  side  effect,  the  framework  also  cleanly  addresses  priority  in¬ 
version  by  providing  a  generalized  form  of  priority  inheritance  that  automatically  works  within  and  among 
diverse  scheduling  policies.  CPU  inheritance  scheduling  extends  naturally  to  multiprocessors,  and  supports 
processor  management  techniques  such  as  processor  affinity  and  scheduler  activations.  In  our  prototype  im¬ 
plemented  we  showed  that  this  flexibility  can  be  provided  with  acceptable  overhead  in  typical  environments, 
depending  on  factors  such  as  context  switch  speed  and  frequency. 

2.1.6  The  Fluke  Device  Driver  Framework 

Providing  efficient  device  driver  support  in  the  Fluke  operating  system  presents  novel  challenges,  which  stem 
from  two  conflicting  factors:  (i)  for  maintainance  and  economic  reasons,  a  requirement  to  reuse  unmodified 
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legacy  device  drivers,  as  encapsulated  in  the  OSKit,  discussed  later  in  Section  2.2,  and  (ii)  the  mismatch 
between  the  Fluke  kernel  s  internal  execution  environment  and  the  execution  environment  expected  by  these 
legacy  device  drivers.  In  this  work,  which  was  documented  in  a  thesis  [6],  we  developed  a  solution  to  this 
conflict:  a  framework  whose  design  is  based  on  running  device  drivers  as  user-mode  servers,  which  resolves 
the  fundamental  execution  environment  mismatch. 

This  approach  introduces  new  problems  and  issues,  of  which  the  most  important  are  synchronization, 
interrupt  delivery,  physical  memory  allocation,  access  to  shared  resources,  and  performance.  We  success¬ 
fully  addressed  the  functional  issues,  as  demonstrated  by  the  fact  that  the  majority  of  device  drivers  execute 
successfully  without  change  and  are  routinely  used  by  Fluke  developers.  Based  on  our  experience  with  the 
minority  of  drivers  that  did  require  changes,  and  our  experience  developing  the  framework,  we  proposed 
guidelines  for  improving  device  drivers’  portability  across  different  execution  environments. 

Running  device  drivers  in  user  mode  raises  serious  performance  issues  but  on  the  whole  they  were  suc¬ 
cessfully  mitigated.  We  compared  the  driver  performance  in  Fluke  with  that  in  the  original  legacy  systems, 
in  terms  of  latency,  bandwidth,  and  processor  utilization.  We  found  that  reasonable  performance  (between 
88—93%  of  the  best-performing  Unix  systems  in  a  realistic  workload)  and  acceptable  processor  overhead 
(between  0-100%)  are  achievable.  The  limiting  factor  is  the  1PC  performance  of  the  underlying  Fluke  OS 
layers. 


2.1.7  Extending  Fluke  IPC  for  Dransparent  Remote  Communication 

Fluke  not  only  provides  a  base  for  the  POSIX-like  environment  that  we  implemented,  it  provides  a  base  for 
experimental  OS  features.  One  such  feature  is  the  checkpointer  described  above.  In  a  major  effort  we  ex¬ 
plored  the  issues  of  providing  transparently  distributed  interprocess  communication  [7]  for  Fluke.  We  had 
several  motivations  for  this  work. 

Distributed  systems  such  as  client-server  applications  and  cluster-based  parallel  computation  are  an  im¬ 
portant  part  of  modem  computing.  Distributed  computing  allows  the  balancing  of  processing  load,  increases 
program  modularity,  isolates  functionality,  and  can  provide  an  element  of  fault  tolerance.  In  these  envi¬ 
ronments,  systems  must  be  able  to  synchronize  and  share  data  through  some  mechanism  for  remote  inter¬ 
process  communication  (IPC).  Although  distributed  systems  have  many  advantages,  they  also  pose  several 
challenges.  One  important  challenge  is  transparency.  It  is  desirable  that  applications  can  be  written  to  a  com¬ 
munication  interface  that  hides  the  details  of  distribution. 

One  way  to  achieve  transparency  is  through  the  extension  of  local  communication  mechanisms  over  a 
network  for  remote  communication.  The  ability  to  transparently  extend  local  communication  depends  on  the 
semantics  of  the  local  IPC  mechanisms.  Unfortunately,  those  semantics  are  often  driven  by  other  architec¬ 
tural  goals  of  the  system  and  may  not  necessarily  be  best  suited  for  remote  communication. 

In  this  work  we  developed  a  remote  IPC  implementation  for  the  Fluke  operating  system  and  documented 
our  findings  in  a  thesis  [7].  We  performed  an  extensive  analysis  of  the  Fluke  architecture,  IPC  system,  and 
IPC  semantics,  with  regard  to  the  extension  of  local  IPC  for  transparent  remote  communication.  We  showed 
that  the  overall  complexity  of  both  the  kernel  IPC  subsystem  and  the  network  IPC  implementation  is  consid¬ 
erably  less  than  similar  operating  systems’  IPC  mechanisms,  and  that  the  Fluke  IPC  architecture  is  generally 
well-suited  for  transparent  remote  IPC.  However,  our  work  also  showed  that  the  lack  of  kernel-provided  ref¬ 
erence  counting  caused  more  problems  than  it  solved,  and  that  the  generality  of  an  important  Fluke  kernel 
object,  the  ‘reference,’  makes  it  impossible  for  the  network  IPC  system  to  provide  completely  transparent 
remote  IPC  without  extensive  additional  services. 
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2.1.8  The  Flask  Security  Architecture:  System  Support  for  Diverse  Security  Policies 

For  a  secure  operating  system  to  be  practical  and  cost  effective  to  build  and  maintain,  it  must  be  useful  to 
a  wide  variety  of  market  segments  and  application  domains.  Therefore  a  single  OS  must  provide  sufficient 
underlying  mechanisms  to  support  the  wide  variety  of  real-world  security  policies,  which  need  to  be  separable 
from  the  OS. 

Such  flexibility  requires  controlling  the  propagation  of  access  rights,  enforcing  fine-grained  access  rights 
and  supporting  the  revocation  of  previously  granted  access  rights.  Previous  systems  are  lacking  in  at  least 
one  of  these  areas.  In  a  technical  report  and  later  a  published  paper  [8],  jointly  written  with  our  NS  A  and  SCC 
collaborators,  we  presented  and  evaluated  the  Flask  operating  system  security  architecture  that  solves  these 
problems.  Control  over  propagation  is  provided  by  ensuring  that  the  security  policy  is  consulted  for  every 
security  decision.  This  control  is  achieved  without  significant  performance  degradation  through  the  use  of  a 
security  decision  caching  mechanism  that  ensures  a  consistent  view  of  policy  decisions.  Both  fine-grained 
access  rights  and  revocation  support  are  provided  by  mechanisms  that  are  directly  integrated  into  the  service¬ 
providing  components  of  the  system.  The  architecture  was  described  through  its  prototype  implementation  in 
the  Flask  microkernel-based  operating  system,  and  the  policy  flexibility  of  the  prototype  was  evaluated.  We 
presented  initial  evidence  that  the  architecture’s  impact  on  both  performance  and  code  complexity  is  modest. 
Moreover,  our  architecture  is  applicable  to  many  other  types  of  operating  systems  and  environments. 

2.1.9  Implementing  Mandatory  Network  Security  in  a  Policy-flexible  System 

The  use  of  networks  is  growing  continuously,  constantly  increasing  the  vulnerability  of  the  computer  sys¬ 
tems  that  use  them.  Current  solutions  for  network  security,  such  as  firewalls,  cannot  support  sophisticated 
trust  relationships  with  external  entities  and  lack  a  comprehensive  approach  to  security.  Research  in  security 
has  shown  the  usefulness  of  mandatory  security  mechanisms  for  supporting  sophisticated  trust  relationships 
and  secure  endpoints  in  addition  to  secure  communication  channels.  Other  efforts  at  incorporating  manda¬ 
tory  security  mechanisms  into  the  network  stack  have  a  limited  notion  of  access  control  policies.  This  work, 
documented  in  a  thesis  [9],  dealt  with  the  design  and  implementation  of  a  more  comprehensive  and  flexible 
network  security  architecture  that  enforces  a  mandatory  access  control  policy  on  network-related  operations 
and  a  mandatory  cryptographic  policy  on  network  traffic. 

The  implementation  involved  modifying  the  FreeBSD  TCP/IP  stack  within  the  Flask  secure  operating 
system.  Access  control  decisions  are  made  in  a  policy-flexible  manner  by  consulting  a  security  server  and 
security  attributes  are  interpreted  only  by  the  security  server.  The  access  control  design  maps  access  control 
requirements  to  checks  made  at  different  layers  in  the  network  stack  according  to  the  functionality  provided 
by  the  layer.  This  approach  has  several  advantages,  which  include  less  time  spent  on  illegal  packets  and  the 
ability  to  specify  policy  in  a  fine-grain  manner.  Network  cryptographic  protection  was  provided  using  the 
IPsec  protocol  for  cryptographic  support  and  the  ISAKMP  protocol  for  key  management. 

2.1.10  Utah  Assurance  Work:  Applying  Formal  Methods  to  Fluke  IPC 

The  formal  methods  community  has  long  known  about  the  need  to  formally  analyze  concurrent  software,  but 
the  OS  community  has  been  slow  to  adopt  such  methods.  The  foremost  reasons  for  this  are  the  cultural  and 
knowledge  gaps  between  formalists  and  OS  hackers,  fostered  by  three  beliefs:  inaccessibility  of  the  tools,  the 
disabling  gap  between  the  validated  model  and  actual  implementation,  and  the  intractable  size  of  operating 
systems.  In  a  paper  at  a  prestigious  and  high  visibility  OS  workshop  [10],  we  showed  these  beliefs  to  be 
untrue  for  appropriately  structured  operating  systems. 
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We  applied  formal  methods  to  verify  properties  of  the  implementation  of  the  Fluke  microkernel's  IP C 
subsystem,  a  major  component  of  the  kernel — over  20%.  In  particular,  we  verified,  in  many  scenarios,  the 
lack  of  deadlock  and  certain  liveness  properties,  with  results  that  apply  to  both  SMP  and  uniprocessor  envi¬ 
ronments.  The  SPIN  model  checker  developed  by  Holzmann  provided  an  exhaustive  concurrency  analysis 
of  the  IPC  subsystem,  unattainable  through  traditional  OS  testing  methods.  In  a  novel  result,  we  found  that 
the  stylized  coding  used  in  Fluke  made  it  relatively  straightforward  to  generate  the  model  directly  from  the 
implementation,  as  opposed  to  from  the  design,  as  is  commonly  done —  giving  stronger  assurance  guaran¬ 
tees.  SPIN  proved  to  be  easily  accessible  to  programmers  inexperienced  with  formal  methods.  We  presented 
our  results  as  a  starting  point  for  a  more  comprehensive  inclusion  of  formal  methods  in  practical  OS  devel¬ 
opment.  In  addition,  our  software  artifact— Fluke  IPC  implemented  in  a  model  checking  language— was 
publically  distributed. 

2.1.11  External  Assurance  Work:  Formal  Analysis  of  Fluke/Flask 

The  Secure  Computing  Corporation  did  extensive  analysis  of  the  Flask  security  architecture  and  its  design 
and  implementation  in  our  Fluke  microkernel-based  system.  At  times  both  we  and  our  NSA  collaborators 
worked  closely  with  SCC,  sometimes  leading  to  changes  in  the  Fluke  design  and  implementation,  and  always 
leading  to  a  clearer  understanding  by  all  parties.  SCC’s  efforts  culminated  in  three  SCC-authored  reports, 
which  provide  a  solid  analysis  of  many  aspects  of  Fluke/Flask,  and  a  base  for  future  efforts.  (We  list  these 
reports  in  Section  4,  but  of  course  do  not  intend  to  imply  that  we  produced  them.) 

Assurance  in  the  Fluke  Microkernel :  Formal  Security  Policy  Model  [11]  presents  the  formal  model  of  the 
security  control  requirements  for  the  Fluke  microkernel,  including  a  formalization  in  the  PVS  specification 
language.  More  generally,  it  provides  a  security  model  for  the  Fluke  and  Flask  systems,  discussing  the  overall 
security  architecture,  security  requirements  on  the  different  elements  of  the  architecture,  and  the  range  of 
policies  supportable  within  the  architecture. 

Assurance  in  the  Fluke  Microkernel:  Formal  Top-Level  Specification  [12]  contains  formal  specifications 
of  the  access  vector  cache  and  prototype  security  server  for  the  Flask  system,  including  a  formalization  in  the 
PVS  specification  language.  It  also  presents  a  model  for  reasoning  about  the  ability  of  a  computing  system  to 
support  dynamic  security  policies,  and  in  particular  changes  to  the  security  policy  which  restrict  operations 
that  were  previously  allowed. 

Assurance  in  the  Fluke  Microkernel:  Final  Report  [13]  contains  the  final  report  for  SCC  s  Fluke  Assur¬ 
ance”  program.  It  includes  a  summary  of  the  objectives,  accomplishments,  and  outputs  from  the  program, 
and  overviews  of  the  major  technical  areas:  the  security  architecture,  the  Fluke  implementation  of  the  se¬ 
curity  architecture,  the  dynamic  security  policy  study,  innovations  in  the  security  policy  server,  results  of 
the  composition  study,  and  suggestions  for  future  work.  Updates  to  the  previously  published  formal  security 
policy  model  and  top  level  specification  are  also  included. 

2.1 .12  Kernel  Performance 

Major  performance  work  in  Fluke  was  pursued  during  the  entire  project.  Some  of  the 
work  had  interesting  and  sometimes  novel  research  aspects.  For  example,  a  number  of  Fluke  kernel  object 
optimizations  have  been  implemented  which  improve  “critical  path”  operations.  In  a  novel  design,  these  op¬ 
timizations  allow  certain  kernel  operations  to  be  performed  in  user  mode,  i.e.,  without  trapping  to  the  kernel, 
and  without  loss  of  integrity  or  atomicity.  Optimized  operations  include  locking  and  unlocking  of  uncon¬ 
tested  mutexes  as  well  as  certain  state  gathering  operations  on  Fluke  references  (capabilities).  In  early  1998 


11 


we  increased  our  efforts  to  improve  performance  in  the  base  system.  In  addition  to  “gprof  ”  (profiling)  sup¬ 
port  for  the  kernel,  we  added  gprof  support  for  user  mode  applications.  Together,  this  support  provided  a 
vital  tool  for  understanding  and  analyzing  the  behavior  of  Fluke  and  its  applications.  Profiling  identified  a 
number  of  problem  areas  which  have  been  corrected  including  inefficient  memory  and  object  allocation  in 
Flick  stubs  and  the  need  for  a  specialized  memory  allocator  for  Fluke  kernel  objects. 

As  another  example,  we  later  designed  and  prototyped  two  IPC  extensions  (atomic  calls  to  perform  “receive- 
message-and-acquire-mutex”  and  “receive-message-with-sender-identification”)  that  enable  a  more  efficient 
implementation  of  the  higher-level  MOM  (see  Section  2.4)  client/server  infrastructure.  Though  simple  in 
concept,  the  extensions  required  careful  analysis  to  ensure  that  they  meet  the  atomicity  requirements  of  the 
Fluke  system  as  well  as  the  security  requirements  of  the  Flask  architecture.  Preliminary  results  showed  im¬ 
provements  of  up  to  30%  on  MOM  operations  that  pass  object  references. 

22  The  OSKit:  Reusable  Components  for  OS  Implementation 

We  gradually  isolated,  formalized,  and  generalized  the  infrastructure  that  we  had  originally  developed  only  to 
support  Fluke.  The  OSKit — reusable  components  for  low-level  systems — went  from  a  rough  prototype  to  a 
robust  set  of  over  30  components  with  carefully  designed  interfaces.  The  OSKit  has  proven  to  be  highly  use¬ 
ful  to  a  wide  variety  of  external  research,  development,  and  even  commercial  projects.  It  currently  supports 
two  architectures,  the  Intel  x86,  important  because  of  its  wide  use,  and  the  StmngARM,  important  because 
of  its  growing  popularity  in  embedded  systems,  due  to  its  combination  of  low  power  and  high  performance. 

Implementing  new  operating  systems  is  tedious,  costly,  and  often  impractical  except  for  large  projects. 
The  Flux  OSKit  addresses  this  problem  in  a  novel  way  by  providing  clean,  well-documented  OS  compo¬ 
nents  designed  to  be  reused  in  a  wide  variety  of  other  environments,  rather  than  defining  a  new  OS  structure. 
The  OSKit  uses  unconventional  techniques  to  maximize  its  usefulness,  such  as  intentionally  exposing  imple¬ 
mentation  details  and  platform-specific  facilities.  Further,  the  OSKit  demonstrates  a  technique  that  allows 
unmodified  code  from  existing  mature  operating  systems  to  be  incorporated  quickly  and  updated  regularly, 
by  wrapping  it  with  a  small  amount  of  carefully  designed  “glue”  code  to  isolate  its  dependencies  and  export 
well-defined  interfaces.  The  OSKit  uses  this  technique  to  incorporate  over  230,000  lines  of  stable  code  in¬ 
cluding  device  drivers,  file  systems,  and  network  protocols.  Our  experience  demonstrates  that  this  approach 
to  component  software  structure  and  reuse  has  a  surprisingly  large  impact  in  the  OS  implementation  domain. 

In  two  papers  [14, 15]  we  described  the  OSKit  along  with  four  real-world  examples  that  showed  how  the  OS¬ 
Kit  is  catalyzing  research  and  development  in  operating  systems  and  programming  languages. 

We  wrote  a  500  page  detailed  manual  [16]  for  the  OSKit,  targeted  primarily  at  users  but  also  containing 
information  on  OSKit  internals.  It  includes  much  expository  text  (rationale,  extended  descriptions),  which  is 
appropriate  as  overview,  background,  and  introductory  information.  The  more  concise  but  extensive  “man 
pages”  constitute  an  API  reference  manual. 

We  made  several  public  releases,  which  were  downloaded  at  a  rate  of  about  1000/month,  including  down¬ 
loads  by  a  number  of  Web  mirror  sites,  so  the  actual  number  of  distributed  copies  will  be  higher.  We  have 
over  470  users  on  the  OSKit  mailing  list. 

Along  with  the  OSKit,  we  released  a  set  of  changes  that  can  be  applied  to  the  Kaffe  OpenVM  (a  popu¬ 
lar,  commercial  open  source  Java  virtual  machine),  enabling  Kaffe  to  link  with  the  OSKit  and  run  on  “bare 
hardware.”  Our  changes  were  integrated  into  the  Kaffe  code  base. 

As  a  step  toward  making  Flask  technology  more  accessible,  we  started  to  integrate  Flask-enhanced  OS¬ 
Kit  components  into  Linux.  In  particular,  we  glued  a  simple  OSKit  filesystem  component  into  Linux  via  a 
dynamically  loadable  kernel  module  (LKM). 
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2.2.1  OSKit  Components 

The  facilities  provided  by  the  OSKit  are  organized  into  two  main  categories,  interfaces  and  libraries. 

Interfaces 

The  OSKit’s  interfaces  are  a  set  of  clean,  object-oriented  interfaces  specified  in  the  framework  of  Microsoft’s 

Component  Object  Model  (COM).  For  example,  the  OSKit  provides  a  “block  I/O  interface  for  communi¬ 
cation  between  file  systems  and  disk  device  drivers,  a  “network  I/O”  interface  for  communication  between 

network  device  drivers  and  protocol  stacks,  and  a  file  system  interface  similar  to  the  “VFS”  interface  in  BSD. 

These  interfaces  are  used  and  shared  by  the  various  OSKit  components  in  order  to  provide  consistency. 

Libraries 

Following  is  a  summary  of  the  34  libraries  currently  provided  by  the  OSKit. 

•  Function  libraries 

-  c:  A  simple,  minimal  C  library  designed  to  work  in  a  restricted  OS  environment. 

-  kern :  Low-level  kernel  support  code  of  all  kinds. 

-  snip:  Kernel  code  providing  symmetric  multiprocessor  support. 

-  com:  Utility  functions  and  wrappers  for  handling  COM  interfaces. 

-  osenv:  Default  implementations  of  the  “glue”  functions  required  by  “large”  encapsulated  com¬ 
ponents  (e.g.,  device  drivers,  networking,  filesystems)  imported  from  other  operating  systems. 

•  POSIX  emulation  and  libraries 

-  posix:  Support  for  what  a  POSIX  conformant  system  would  typically  implement  as  system  calls. 

-  freebsdx :  Complete  POSIX-like  C  library  derived  from  FreeBSD,  providing  both  single-  and 
multithreaded  configurations.  Together  with  the  above  posix  library,  provides  a  very  large  sub¬ 
set  of  the  POSIX  API. 

-  freebsdsn:  Complete  standard  math  library,  taken  from  FreeBSD’s  libm. 

-  fsnamespace :  Provides  file  naming  translation  (“namei”-style),  as  well  as  high  level  mount  and 
unmount  capabilities. 

-  rtld\  Runtime  Link  er/ Loade  r  pro  vide  s  dynamic  linking,  loading,  and  shared  library  facilities. 

•  Memory  management 

-  lmm\  A  flexible  memory  management  library  that  can  manage  either  physical  or  virtual  memory. 
This  library  supports  many  special  features  needed  by  OS-level  code,  such  as  multiple  memory 
types,  allocation  priorities,  and  arbitrary  alignment  and  placement  constraints. 

-  amm:  The  Address  Map  Manager  library  manages  collections  of  resources  where  each  element 
has  a  name  (address)  and  some  set  of  attributes.  Examples  of  resources  include  swap  space  and 
process  virtual  address  space. 
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-  svm:  The  Simple  Virtual  Memory  library  uses  the  AMM  library  to  define  a  simple  virtual-memory 
interface  for  a  single  address  space  that  can  provide  memory  protection  and  paging  to  a  block 
device. 

•  Threads,  synchronization,  and  scheduling 

-  threads:  This  library  provides  support  for  multithreaded  kernels,  including  POSIX  threads,  syn¬ 
chronization,  scheduling,  and  stack  guards.  Scheduling  policies  are  the  standard  posix  Round- 
Robin  and  and  FIFO,  as  well  as  the  CPU  inheritance  scheduling  framework  with  provides  rate- 
monotonic,  stride  (WFQ),  and  lottery  scheduling  policies. 

•  Development  aids 

-  memdebug:  Provides  debugging  versions  of  malloc  et  al. 

-  gprof.  Run-time  profiling  of  kernels. 

•  Simple  disk/file  reading  and  loading 

-  diskpart:  Recognizes  various  common  disk  partitioning  schemes  and  produces  a  complete  “map” 
of  the  organization  of  any  disk. 

-  fsread:  A  simple  read-only  file  system  interpretation  library  supporting  various  common  types 
of  file  systems;  typically  used  for  bootstrapping. 

-  exec:  A  generic  executable  interpreter  and  loader  that  supports  popular  executable  formats. 

•  Filesystem  implementations 

-  linux/s:  Encapsulated  Linux  filesystem  code.  Includes  support  for  ext2,  the  primary  Linux 
filesystem,  as  well  as  numerous  other  PC  filesystems  supported  under  Linux. 

-  netbsd/s:  Encapsulated  NetBSD  filesystem  code. 

-  memfs:  A  trivial  memory-based  filesystem  (i.e.,  a  RAM  disk),  exporting  the  standard  OSKit 
filesystem  interfaces. 

•  Networking  implementations 

-  freebsdjiet:  Encapsulated  FreeBSD  networking  code,  including  sockets. 

-  bootp:  BOOTP  protocol  (RFC  1048/1533)  support. 

-  hpfq:  Hierarchical  proportional-share  control  of  outgoing  network  link  bandwidth. 

-  dpf:  A  packet  dispatching  mechanism  using  packet  filter  technology. 

•  Device  driver  implementations 

-  linuxJev:  Encapsulated  Linux  device  drivers.  Currently  includes  over  50  block  (SCSI,  IDE)  and 
network  drivers. 

-  freebsd-dev:  Encapsulated  FreeBSD  device  driver  set.  Currently  includes  eight  TTY  (virtual 
console  and  serial  line,  including  mouse)  drivers. 

•  Video  and  window  manager  implementations 

-  wimpi:  Simple  hierarchical  windowing  system. 
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-  *vicleo *:  Basic  video  support,  with  two  implementations:  one  encapsulating  all  of  SVGALIB 
with  broad  device  support,  and  one  based  on  XFree86,  but  with  only  the  S3  driver  currently  sup¬ 
ported.  We  also  provide  support  for  XI 1  clients. 

•  Miscellaneous 

The  following  four  libraries  are  are  not  as  well  documented  but  are  still  very  useful,  especially  the  first 
two. 

-  unix :  Support  to  debug  and  run  many  OSKit  components  on  FreeBSD  and  Linux  in  user-mode. 
Very  useful  for  debugging. 

-  startup :  Contains  functions  to  start  up  and  initialize  various  OSKit  components. 

-  fudp :  Provides  a  “Fake  UDP”  implementation:  a  restricted  send-only  no-fragmenting  version  of 
UDP. 

-  unsupp :  Contains  various  unsupported  hacks  and  utilities. 

2.3  The  Flick  Interface  Definition  Language  Compiler 

In  our  work  to  optimize  Mach  and,  later,  Fluke  IPC,  we  found  it  essential  to  improve  the  user-space  costs  in¬ 
volved,  as  well  as  to  achieve  more  flexibility.  A  major  source  of  costs  proved  to  be  the  generation  of  stub 
code  from  the  interface  specification  in  IDL. 

2.3.1  General,  Flexible,  and  Optimizing  IDL  Compilation 

An  interface  definition  language  (DDL)  is  a  nontraditional  language  for  describing  interfaces  between  soft¬ 
ware  components.  DDL  compilers  generate  “stubs”  that  provide  separate  communicating  processes  with  the 
abstraction  of  local  object  invocation  or  procedure  call.  High-quality  stub  generation  is  essential  for  appli¬ 
cations  to  benefit  from  component-based  designs,  whether  the  components  reside  on  a  single  computer  or  on 
multiple  networked  hosts.  Typical  IDL  compilers,  however,  do  little  code  optimization,  incorrectly  assum¬ 
ing  that  interprocess  communication  is  always  the  primary  bottleneck.  (As  networks  and  operating  systems 
become  faster,  the  bottleneck  for  structured  communication  (RPC,  object  invocation)  will  move  to  the  pre¬ 
sentation  layer.)  More  generally,  typical  DDL  compilers  are  “rigid”  and  limited  to  supporting  only  a  single 
DDL,  a  fixed  mapping  onto  a  target  language,  and  a  narrow  range  of  data  encodings  and  transport  mechanisms. 

Flick ,  our  DDL  compiler,  is  based  on  the  insight  that  DDLs  are  true  languages  amenable  to  modem  com¬ 
pilation  techniques.  Flick  exploits  concepts  from  traditional  programming  language  compilers  to  bring  both 
flexibility  and  optimization  to  the  domain  of  FDL  compilation.  Through  the  use  of  carefully  chosen  inter¬ 
mediate  representations,  Flick  supports  multiple  DDLs,  diverse  data  encodings,  multiple  transport  mecha¬ 
nisms,  and  applies  numerous  optimizations  to  all  of  the  code  it  generates.  Our  experiments  showed  that 
Flick-generated  stubs  marshal  data  between  2  and  17  times  faster  than  stubs  produced  by  traditional  IDL 
compilers,  and  on  today’s  generic  operating  systems,  increased  end-to-end  throughput  by  factors  between 
1.2  and  3.7. 

Our  primary  paper  on  Flick  [17]  was  published  in  the  most  prestigious  and  competitive  venue  for  com¬ 
piler  research,  the  ACM  SIGPLAN  Conference  on  Programming  Language  Design  and  Implementation. 
However,  Flick  was  not  merely  a  research  advance;  it  became  a  robust,  well  supported,  well  documented 
tool.  During  the  course  of  the  project  we  made  several  formal,  public  releases  of  Flick.  The  releases  in¬ 
cluded  a  user’s  manual  [18]  that  describes  how  to  build  Flick,  how  to  run  the  various  compiler  passes,  and 
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how  to  use  the  generated  stubs.  The  manual  presents  a  simple  client/server  phonebook,  implemented  in  three 
different  ways:  as  an  ONC  RPC  (Sun  RPC)  application,  as  a  CORBA  C  application,  and  as  a  CORBA  C++ 
application.  The  examples  are  presented  in  detail  and  illustrate  how  users  can  make  use  of  Flick-generated 
stubs  in  their  own  programs. 

We  also  produced  a  Flick  internals  manual  [19],  which  describes  its  implementation  details.  People  in¬ 
terested  in  studying,  modifying,  or  extending  Flick  should  consult  this  manual  as  a  starting  point.  Those  who 
simply  want  to  use  Flick  do  not  need  to  read  this  document:  use  of  the  Flick  compiler  tools  is  described  in 
the  separate  user’s  manual. 

Some  of  these  releases  contained  dramatic  improvements  in  function,  completeness,  and  performance. 
New  releases  supported  new  transports  and  encodings,  including  Fluke  1PC,  CORBA  HOP,  Sun  ONC,  and 
Trapeze  (a  gigabit  Myrinet-based  communication  protocol  from  Duke  University).  Flick  supported  new 
IDLs  and  message  formats,  including  almost  complete  support  for  Mach  MIG,  and  support  for  security  anno¬ 
tations  on  interfaces.  New  releases  supported  new  platforms,  including  Windows  95/NT.  Support  for  CORBA 
improved  significantly:  we  supported  CORBA  on  Mach  and  provided  a  greatly  improved  CORBA  runtime 
during  the  course  of  the  project. 

Because  Flick  provides  a  unique  combination  of  flexibility  and  optimization.  Flick  was  chosen  to  be  part 
of  the  Quorum  Distributed  Objects  Integration  (DOI)  project.  The  initial  integration  efforts  went  well:  for 
instance,  in  1998  Joe  Loyall  of  BBN  successfully  modified  BBN’s  QuO  “delegate  generator”  to  work  with 
Flick’s  CORBA  IDL  front  end.  We  worked  with  the  Quorum  distributed  object  integration  team,  in  par¬ 
ticular  BBN  and  Washington  University  at  St.  Louis,  to  evolve  Flick  to  support  their  needs,  in  particular 
aHHing  complete  C++  support  and  support  for  WUSTL’s  TAO  system.  Our  implementation  of  the  CORBA 
C++  mapping  resulted  in  many  additions  to  Flick,  because  the  CORBA  C++  mapping  is  significantly  more 
complex  than  any  of  the  other  (C  language)  mappings  we  had  previously  implemented. 

2 32  Flexible  IDL  Compilation  for  Complex  Communication  Patterns 

We  continued  to  develop  Flick  for  other  uses  as  well.  In  particular,  we  developed  a  strategy  for  using  Flick 
to  generate  stubs  for  Khazana,  our  global  memory  service  requiring  asynchronous  messages  between  nodes, 
described  in  Section  2.6.  Flick  now  produces  specially  “decomposed”  CORBA  stubs,  achieving  flexible  IDL 
compilation  for  complex  communication  patterns. 

Distributed  applications  are  complex  by  nature,  so  it  is  essential  that  there  be  effective  software  devel¬ 
opment  tools  to  aid  in  the  construction  of  these  programs.  Commonplace  “middleware”  tools,  however,  of¬ 
ten  impose  a  tradeoff  between  programmer  productivity  and  application  performance.  For  instance,  many 
CORBA  IDL  compilers  generate  code  that  is  too  slow  for  high-performance  systems.  More  importantly, 
these  compilers  provide  inadequate  support  for  sophisticated  patterns  of  communication.  We  believe  that 
these  problems  can  be  overcome,  thus  making  IDL  compilers  and  similar  middleware  tools  useful  for  a  broader 
range  of  systems. 

To  this  end  we  extended  Flick  to  produce  specialized  high-performance  code  for  complex  distributed 
applications.  Flick  can  produce  specially  “decomposed”  stubs  that  encapsulate  different  aspects  of  com¬ 
munication  in  separate  functions,  thus  providing  application  programmers  with  fine-grain  control  over  all 
messages.  The  design  of  our  decomposed  stubs  was  inspired  by  the  requirements  of  a  particular  distributed 
application  called  Khazana.  In  two  papers  [20,  21]  we  described  our  experience  in  refitting  Khazana  with 
Flick-generated  stubs.  We  believe  that  the  special  IDL  compilation  techniques  developed  for  Khazana  will 
be  useful  in  other  applications  with  similar  communication  requirements. 
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2.4  MOM:  The  Mini  Object  Model 


We  designed,  specified  [22],  and  implemented  MOM,  the  “Mini-Object  Model”:  a  layer  above  Fluke  IPC 
that  presents  a  more  client/server-oriented  interface  to  applications.  MOM  specifies  a  simple  object  invoca¬ 
tion  model  and  language-specific  bindings  to  the  facilities  of  that  model.  The  purpose  of  this  specification 
is  to  facilitate  the  portability  of  modules  written  for  message-passing  interfaces  to  a  variety  of  operating  en¬ 
vironments.  Many  components  implemented  in  client-server  systems  using  microkernel  IPC  facilities,  in 
complex  client-server  infrastructure  systems  such  as  COM  and  CORBA,  and  in  subsystems  of  monolithic 
operating  system  kernels,  in  fact  assume  only  a  conceptually  simple  model  of  object  invocation  and  rely  on 
relatively  few  other  incidental  operating  system  facilities.  This  specification  hopes  to  provide  an  interface 
for  object  management  and  invocation  that  can  be  implemented  in  a  variety  of  operating  environments  and 
impose  little  or  no  run-time  overhead  on  modules  using  this  interface  instead  of  the  specific  interface  native 
to  a  particular  IPC  system,  OS  kernel,  or  complex  object  system. 

During  our  analysis  and  performance  improvements  to  the  Fluke  Operating  System  a  number  of  exper¬ 
iments  identified  our  MOM  infrastructure  as  a  problem  area.  To  address  this  problem,  we  made  significant 
algorithmic  improvements  to  the  reference  counting  component  of  MOM,  in  addition  to  making  other  MOM 
performance  enhancements.  We  created  a  standalone  benchmark  to  assist  in  tracking  the  effect  of  future 
changes  to  MOM. 

2.5  Java  Operating  Systems 

In  the  latter  parts  of  the  contract  periods  we  explored  the  implications  of  applying  our  OS  structuring  ideas 
to  operating  systems  that  use  the  type-safe  properties  of  the  Java  programming  language  to  provide  memory 
safety,  instead  of  using  the  hardware  MMU  to  do  so.  We  explored  many  aspects  of  the  issues  in  two  exper¬ 
imental  prototypes,  with  an  emphasis  on  resource  control.  In  our  largest  effort  in  this  domain,  we  designed 
and  implemented  “Alta,”  an  implementation  of  the  Fluke  nested  process  model  in  a  Java  virtual  machine. 

2.5.1  Making  the  Case:  OS  Structure  for  Mobile  Code 

In  a  highly  competitive  and  visible  OS  workshop,  we  presented  and  published  early  results  from  our  Alta 
prototype  [23],  focusing  on  making  the  case  for  such  a  comprehensive  approach.  The  majority  of  work  on 
protection  in  single-language  mobile  code  environments  focuses  on  information  security  issues  and  depends 
on  the  language  environment  for  solutions  to  the  problems  of  resource  management  and  process  isolation. 
We  argued  that  what  is  needed  in  these  environments  are  not  ad-hoc  or  incremental  changes  but  a  coherent 
approach  to  security,  failure  isolation,  and  resource  management.  Protection,  separation,  and  control  of  the 
resources  used  by  mutually  untrusting  components,  applets,  applications,  or  agents  are  exactly  prob¬ 

lems  faced  by  multi-useroperating  systems.  We  argued  that  real  solutions  will  come  only  if  an  OS  model  is 
uniformly  applied  to  these  environments.  We  presented  Alta,  our  prototype  Java-based  system  patterned  on 
Fluke,  a  highly  structured,  hardware-based  OS,  and  reported  on  its  features  appropriate  to  controlling  mobile 
code. 


2.5.2  The  Alta  Operating  System 

Many  modem  systems,  including  web  servers,  database  engines,  and  operating  system  kernels,  are  using 
language-based  protection  mechanisms  to  provide  the  safety  and  integrity  traditionally  supplied  by  hardware. 
As  these  language-based  systems  become  used  in  more  demanding  situations,  they  are  faced  with  die  same 
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problems  that  traditional  operating  systems  have  solved — namely  shared  resource  management,  process  sep¬ 
aration,  and  per-process  resource  accounting.  While  many  incremental  changes  to  language-based,  extensi¬ 
ble  systems  have  been  proposed,  we  demonstrated  in  an  implementation  and  a  thesis  [24]  that  comprehensive 
solutions  used  in  traditional  operating  systems  are  applicable  and  appropriate. 

The  thesis  gives  a  detailed  description  of  Alta,  an  implementation  of  the  Fluke  operating  system’s  nested 
process  model  in  a  Java  virtual  machine.  The  nested  process  model  is  a  hierarchical  operating  system  process 
model  designed  to  provide  a  consistent  approach  to  user-level,  per-process  resource  accounting  and  control. 
This  model  accounts  for  CPU  usage,  memory,  and  other  resources  through  a  combination  of  system  primi¬ 
tives  and  a  flexible,  capability-based  mechanism. 

Alta  supports  nested  processes  and  interprocess  communication.  Java  applications  running  on  Alta  can 
create  child  processes  and  regulate  the  resources — the  environment — of  those  processes.  Alta  demonstrates 
that  the  Java  environment  is  sufficient  for  hosting  traditional  operating  system  abstractions.  Alta  extends  the 
nested  process  model  to  encompass  Java-specific  resources  such  as  class  files,  modifies  the  model  to  leverage 
Java’s  type  safety,  and  extends  the  Java  type  system  to  support  safe  fine-grained  sharing  between  different 
applications.  Existing  Java  applications  work  without  modification  on  Alta. 

Alta  was  compared  in  terms  of  structure,  implementation  and  performance  to  Fluke  and  traditional  hardware- 
based  operating  systems.  A  small  set  of  test  applications  demonstrated  flexible,  application-level  control  over 
memory  usage  and  file  access. 

2.5.3  Java  Operating  Systems:  Comparative  Design  and  Implementation 

Language-based  extensible  systems  such  as  Java  use  type  safety  to  provide  memory  safety  in  a  single  address 
space.  Memory  safety  alone,  however,  is  not  sufficient  to  protect  different  applications  from  each  other.  Such 
systems  must  support  a  process  model  that  enables  the  control  and  management  of  computational  resources. 

In  particular,  language-based  extensible  systems  must  support  resource  control  mechanisms  analogous  to 
those  in  standard  operating  systems.  They  must  support  the  separation  of  processes  and  limit  their  use  of 
resources,  but  still  support  safe  and  efficient  interprocess  communication. 

In  our  prototypes  and  in  a  paper  [25]  we  demonstrated  how  this  challenge  can  be  addressed  in  Java  oper- 
ating  systems.  First,  we  described  the  technical  issues  that  arise  when  implementing  a  process  model  in  Java. 

In  particular,  we  laid  out  the  design  choices  for  managing  resources.  Second,  we  described  the  solutions  that 
we  explored  in  two  complementary  projects,  Alta  and  GVM.  GVM  is  similar  to  a  traditional  monolithic  ker¬ 
nel,  whereas  Alta  closely  models  the  Fluke  operating  system.  Features  of  our  prototypes  include  flexible 
control  of  processor  time  using  CPU  inheritance  scheduling,  per-process  memory  controls,  fair  allocation 
of  network  bandwidth,  and  execution  directly  on  hardware  using  the  OSKit.  Finally,  we  compared  our  pro¬ 
totypes  with  other  language-based  operating  systems,  in  particular  Cornell’s  “J-Kemel,”  and  explored  the 
tradeoffs  between  the  various  designs. 

2.5.4  Drawing  the  Red  Line  in  Java 

Software-based  protection  has  become  a  viable  alternative  to  hardware-based  protection  in  systems  based 
on  languages  such  as  Java,  but  the  absence  of  hardware  mechanisms  for  protection  has  been  coupled  with 
an  absence  of  a  user/kemel  boundary.  In  a  well-received  paper  [26]  we  showed  why  such  a  “red  line”  must 
be  present  in  order  for  a  Java  virtual  machine  to  be  as  effective  and  as  reliable  as  an  operating  system.  We 
discussed  how  the  red  line  can  be  implemented  using  software  mechanisms,  and  explained  the  ones  we  use 
in  the  Java  system  that  we  are  building. 
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2.5.5  Memory  Management:  The  Need  for  Predictable  Garbage  Collection 

Modem  programming  languages  such  as  Java  are  increasingly  being  used  to  write  systems  programs.  By 
systems  programs,  we  mean  programs  that  provide  critical  services  (compilers),  are  long-running  (Web 
servers),  or  have  time-critical  aspects  (databases  or  query  engines).  One  of  the  requirements  of  such  pro¬ 
grams  is  predictable  behavior.  Unfortunately,  predictability  is  often  compromised  by  the  presence  of  garbage 
collection.  Various  researchers  have  examined  the  feasibility  of  replacing  garbage  collection  with  forms  of 
stack  allocation  that  are  more  predictable  than  GC,  but  the  applicability  of  such  research  to  systems  programs 
had  not  previously  been  studied  or  measured.  A  particularly  promising  approach  allocates  objects  in  the  nth 
stack  frame  (instead  of  just  the  topmost  frame):  we  call  this  deep  stack  allocation.  In  a  detailed  pap er  [27]  we 
presented  dynamic  profiling  results  for  several  Java  programs  to  show  that  deep  stack  allocation  shouldbene- 
lit  systems  programs,  and  we  described  the  approach  that  we  are  developing  to  perform  deep  stack  allocation 
in  Java. 

2.6  The  Khazana/KOLA  Infrastructure  for  Building  Distributed  Services 

In  1998  we  transitioned  our  DSM  effort  away  from  LAN -based  parallel  numeric  applications  to  a  broader 
and  more  prevalent  class  of  applications:  local  and  wide-area  distributed  applications.  The  base  layerof  our 
new  architecture  is  called  Khazana-,  above  that  are  two  layers  that  comprise  KOLA,  the  Khazana  Object  Layer 
Architecture. 

Description  of  Khazana:  Essentially  all  distributed  systems,  applications,  and  services  at  some  level  boil 
down  to  the  problem  of  managing  distributed  shared  state.  Unfortunately,  while  tire  problem  of  managing  dis¬ 
tributed  shared  state  is  shared  by  many  applications,  there  is  no  common  means  of  managing  the  data— every 
application  devises  its  own  solution.  We  developed  Khazana,  a  distributed  service  exporting  the  abstraction 
of  a  distributed  persistent  globally  shared  store  that  applications  can  use  to  store  their  shared  state  and  de¬ 
scribed  it  in  a  paper  [28].  Khazana  is  responsible  for  performing  many  of  the  common  operations  needed 
by  distributed  applications,  including  replication,  consistency  management,  fault  recovery,  access  control, 
and  location  management.  Using  Khazana  as  a  form  of  middleware,  distributed  applications  can  be  quickly 
developed  from  corresponding  uniprocessor  applications  through  the  insertion  of  Khazana  data  access  and 
synchronization  operations. 

Experience  with  Khazana-based  Applications:  We  evaluated  the  effectiveness  of  basing  distributed  sys¬ 
tems  on  a  persistent  globally  shared  address  space  abstraction,  as  implemented  by  Khazana,  and  documented 
our  results  in  a  paper  [29].  Khazana  provides  shared  state  management  services  to  distributed  application  de¬ 
velopers,  including  consistent  caching,  automated  replication  and  migration  of  data,  location  management, 
access  control,  and  (limited)  fault  tolerance.  We  reported  on  our  experience  porting  three  applications  to 
Khazana:  a  distributed  file  system  (KFS)  based  on  Linux  local  filesystem  code,  a  distributed  name  service 
(KNS),  and  a  cooperative  drawing  program  based  on  xf  ig  (KFIG).  The  basic  Khazana  abstraction  of  persis¬ 
tent  shared  memory  made  it  easy  to  create  fairly  efficient  distributed  services  with  flat  data  abstractions  (KFS 
and  KNS),  but  our  port  of  xf  ig  made  it  clear  that  a  higher  level  abstraction  is  preferable  for  applications 
with  more  structured  state.  As  a  result,  we  extended  Khazana  to  support  a  limited  set  of  object-like  functions 
(reference  swizzling,  event  upcalls,  and  update  propagation).  We  present  herein  the  current  Khazana  design 
and  implementation,  and  discuss  the  lessons  learned  from  our  initial  evaluation  of  it. 
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KOLA:  the  Khazana  Object  Layer  Architecture:  Based  on  our  experience  with  Khazana,  we  designed  a 
form  of  three-tier  middleware  for  distributed  applications  and  services  based  on  Khazana.  The  lowest  layer, 
Khazana  proper,  provides  basic  distributed  storage,  consistency,  availability,  locking,  and  security  mecha¬ 
nisms  for  unstructured  page-oriented  data.  The  middle  layer  provides  language  independent  support  for  “ob¬ 
jects,”  including  object  caching,  persistent  heap  management,  basic  reference  swizzling,  and  an  extension  of 
the  basic  “lock  intents”  mechanism  that  supports  independent  locking  of  multiple  objects  within  a  page.  The 
top  layer  is  a  set  of  language  specific  libraries  that  support  method  invocation  on  remotely  instantiated  ob¬ 
jects,  run  time  type  inference,  language-specific  reference  swizzling,  and  (optional)  garbage  collection.  The 
middle  and  top  layer  are  referred  to  collectively  as  KOLA  (Khazana  Object  Layer  Architecture). 

KOLA  allows  us  to  support  reference-filled,  pointer-rich  applications  while  allowing  the  base  “region- 
based”  Khazana  to  co-exist  independently.  Our  initial  version  of  KOLA  demonstrated  the  feasibility  of  ex¬ 
tracting  the  necessary  programming  language-level  information  to  support  persistent  distributed  objects  on 
top  of  Khazana. 

A  Language-Specific  Layer:  Distributed  Persistent  C++  Objects:  We  designed  and  implemented  a 
C++  object  layer  for  Khazana.  The  C++  layer  described  herein  lets  programmers  use  familiar  C++  idioms  to 
allocate,  manipulate,  and  deallocate  persistent  shared  data  structures.  It  handles  the  tedious  details  involved 
in  accessing  this  shared  data,  replicating  it,  maintaining  consistency,  converting  data  representations  between 
persistent  and  in-memory  representations,  associating  type  information  including  methods  with  objects,  etc. 
To  support  the  C++  object  layer  on  top  of  Khazana’s  flat  storage  abstraction,  we  developed  a  language- 
specific  preprocessor  that  generates  support  code  to  manage  the  user-specified  persistent  C++  structures.  In  a 
paper  [30]  we  described  the  design  of  the  C++  object  layer  and  the  compiler  and  runtime  mechanisms  needed 
to  support  it. 
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3  Technology  Transfer 


Most  technology  transfer  was  mentioned  under  each  software  component  in  the  “Results”  section.  A  few 
aspects  will  be  mentioned  here. 


3.1  Software 

The  OSKit:  The  releases  were  picked  up  by  hundreds  of  sites  and  have  many  users,  ranging  from  the  DARPA- 
funded  Express  project  at  MIT,  which  is  using  the  OSKit  to  develop  an  operating  system  focused  around  the 
ML  implementation  language,  to  Network  Storage  Solutions,  a  company  using  the  OSKit  inside  network- 
attached  “file  service  appliances.”  Using  the  OSKit,  MIT’s  Express  project  has  gotten  CMU’s  Foxnet  system 
(TCP/IP  in  ML)  to  run  on  the  bare  hardware.  Other  users  include  the  DARPA-funded  exokemel  project  at 

MIT/CMU  ,  the  L4-clone-based  Real-time  Operating  Systems  project  atTU-Dresden,  and  the  Rice  University 
Programming  Languages  group. 

Flick  IDL  Compiler  Kit:  robust  releases  were  downloaded  by  over  650  sites  in  1998  alone.  Flick  was 
used  for  the  DARPA-funded  “Porcupine”  distributed,  scalable  mail  service  at  the  University  of  Washington. 
The  real-time  OS  project  at  Technische  Universitat  Dresden  ported  Flick  to  L4  IPC,  and  are  using  it  regu¬ 
larly.  Flick  was  part  of  BBN’s  DARPA-sponsored  QuOIN  2.0  software  release  in  1999,  under  the  Quorum 
distributed  object  integration  project. 

Various  commercial  companies  have  also  expressed  interest  in  using  Flick.  The  most  promising  is  Apple 
We  have  been  talking  regularly  with  Apple  about  the  possibilities  for  using  Hick  to  generate  code  for  their 
Mac  OS  X  operating  system,  because  Hick  offers  the  opportunity  to  help  bridge  the  many  disparate  object 
models  and  implementations  in  their  system.  From  this  interaction  we  have  been  learning  that  Flick  may 
have  much  to  offer  legacy  systems  through  its  flexibility.  In  December  1998,  BBN  included  Hick  in  their 
(unfunded)  proposal  for  DARPA  BAA  99-03,  “Data  Intensive  Computing.” 

Huke/Hask:  We  publically  released  the  Huke/Hask  kernel  source  in  February  1999,  timed  to  coincide 
with  our  OSDI  paper  on  Huke  “atomicity.”  Portions  of  the  Hask  servers  were  released  as  part  of  regular 
OSKit  releases  during  1999.  We  plan  to  release  the  rest  of  the  Huke/Hask  operating  system  later  this  year, 
timed  to  coincide  with  the  release  of  Fluke’s  Java  virtual  machine  incarnation,  “Alta.”  Our  NS  A  collaborators 

are  integrating  the  Hask  architecture  and  existing  components  into  Linux. 

3.2  Personnel 

National  Security  Agency:  two  section  R23  (Information  Security  Research)  researchers,  Jeff  Turner  and 
Steve  Smalley,  worked  on-site  at  the  University  of  Utah  for  one  year  from  August  1996  through  July  1997. 
They  worked  on  Huke/Flask  and  other  security  and  assurance-related  aspects  of  the  Hux  project.  Close  inter¬ 
action  with  these  and  other  R23  personnel  continued  for  the  duration  of  the  project,  and  in  general,  continues 

to  this  day.  Besides  regular  email  and  phone  meetings,  this  includes  joint  paper  writing  and  read/write  access 
to  shared  source  repositories, 

Greg  Benson,  a  Ph.D.  student  in  the  CS  department  at  the  University  of  California  at  Davis,  joined  us  as 
a  visiting  Research  Assistant  from  January-May  1997.  His  thesis  topic  was  “run  time  support  for  advanced 
programming  languages”  and  while  with  us  he  ported  the  SR  parallel  language  to  the  OSKit  He  co-authored 
a  paper  with  us  on  the  OSKit. 

Khazana:  many  of  the  ideas  and  mechanisms  have  involved  two-way  tech  transfer  between  Utah  and 
MangoSoft  Corporation  (http://www.mango.com/),  due  to  the  co-PI’s  consulting  involvement. 
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Our  many  changes  to  Kaffe,  a  popular  commercial  open-source  Java  Virtual  Machine,  to  support  flexible 
multithreading,  flexible  scheduling,  improve  robustness,  and  to  support  the  OSKit,  have  been  incorporated 
into  the  base  version  of  Kaffe. 

33  Relationships 

During  the  course  of  the  grant  periods,  we  forged  relationships  with  many  companies  who  were  interested  in 
the  DARPA-sponsored  technology  we  were  developing.  For  two  companies,  this  interest  was  great  enough 
that  it  was  manifested  in  grants.  To  help  support  the  research  reported  herein,  during  the  grant  period  we 
received  several  cash  grants  from  Novell  and  an  equipment  grant  from  Compaq/DEC.  We  had  numerous 
technical  meetings  with  Novell,  the  Open  Software  Foundation,  Hewlett-Packard,  Sun  Microsystems,  Ap¬ 
ple  Computer,  IBM,  Secure  Computing  Corporation,  Trusted  Information  Systems,  and  BBN,  to  discuss  our 
research  and  its  relevance  to  their  needs. 
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technology  for  aerospace  command  and  control  and  its  transition  to  air, 
space,  and  ground  systems  to  meet  customer  needs  in  the  areas  of  Global 
Awareness,  Dynamic  Planning  and  Execution,  and  Global  Information 
Exchange  is  the  focus  of  this  AFRL  organization.  The  directorate’s  areas 
of  investigation  include  a  broad  spectrum  of  information  and  fusion, 
communication,  collaborative  environment  and  modeling  and  simulation, 
defensive  information  warfare,  and  intelligent  information  systems 
technologies. 


